Performance of an artificial intelligence tool with real-time clinical workflow integration – Detection of intracranial hemorrhage and pulmonary embolism

•Intra-cranial hemorrhage and pulmonary embolism are life-threatening pathologies.•CT imaging is essential to confirm diagnosis.•In a real-time clinical setting AI shows the potential rule out ICH PE.•The positive predictive value of remains moderate.•AI has assist radiologists serve as adjunct. IntroductionAcute pathologies require early detection with prompt communication critical findings ensure adequate management. Intra-cranial (ICH) (PE) two such frequent pathologies, significant morbidity mortality, where misdiagnosis can lead adverse outcome [1van Asch C.J. Luitse M.J. Rinkel G.J. van der Tweel I. Algra A. Klijn Incidence, case fatality, functional intracerebral haemorrhage over time, according age, sex, ethnic origin: systematic review meta-analysis.Lancet Neurol. 2010; 9: 167-176https://doi.org/10.1016/S1474-4422(09)70340-0Abstract Full Text PDF PubMed Scopus (1595) Google Scholar, 2Heit J.J. Iv M. Wintermark Imaging intracranial hemorrhage.J Stroke. 2017; 19: 11-27https://doi.org/10.5853/jos.2016.00563Crossref (99) 3Morales H. Pitfalls in interpretation hemorrhage.Semin Ultrasound CT MR. 2018; 39: 457-468https://doi.org/10.1053/j.sult.2018.07.001Crossref (6) Scholar]. A non-contrast head scan diagnosis risk stratification ICH, while contrast enhanced Computed Tomography Pulmonary Angiography (CTPA) standard for detecting locating PE [3Morales 4Estrada-Y-Martin R.M. Oldham S.A. CTPA gold embolism.Int J Comput Assist Radiol Surg. 2011; 6: 557-563https://doi.org/10.1007/s11548-010-0526-4Crossref (40) Scholar].Advances technology have led improvement image quality reduction radiation dose, which allows more subtle lesions. However, increasing volume number examinations images per examination, disproportionate effect on radiologist’ work stream. McDonald et al., calculated their study influence technological advancements cross-sectional radiology workflow, that radiologist analyses an average one every three seconds [[5]McDonald R.J. Schwartz K.M. Eckel L.J. al.The effects changes utilization workload.Acad Radiol. 2015; 22: 1191-1198https://doi.org/10.1016/j.acra.2015.05.007Abstract (149) This time-intensive encumbrance practicing radiologist, accrue increase false negative results [6Grob D. Smit E. Oostveen al.Image iodine maps embolism: comparison subtraction dual-energy [published online ahead print, 2019 Mar 12].AJR Am Roentgenol. 2019; 1–7https://doi.org/10.2214/AJR.18.20786Crossref (7) 7Brady A.P. Error discrepancy radiology: inevitable or avoidable?.Insights Imaging. 8: 171-182https://doi.org/10.1007/s13244-016-0534-1Crossref (193) 8Sokolovskaya Shinde T. Ruchman R.B. faster reporting speed studies misses errors: pilot study.J Coll 12: 683-688https://doi.org/10.1016/j.jacr.2015.03.040Abstract (53) Real-time double reading by peer often done, been proved aid lowering prevalence misdiagnosis, however it very labor-intensive. In addition, retrospective reviewing cases does not immediate improve patient’s outcome, especially acute [9Geijer Geijer Added diagnostic radiology, review.Insights 287-301https://doi.org/10.1007/s13244-018-0599-0Crossref (39) 10Muroff L.R. Berlin L. Speed versus accuracy: Current thoughts literature review.AJR 213: 490-492https://doi.org/10.2214/AJR.19.21290Crossref (8) 11Babiarz L.S. Yousem D.M. Quality control neuroradiology: discrepancies among academic neuroradiologists.AJNR Neuroradiol. 2012; 33: 37-42https://doi.org/10.3174/ajnr.A2704Crossref (29) Scholar].Given PE, workload, constant development new advanced computed tomography techniques nowadays pandemics our health care system, artificial intelligence (AI) technologies adjunct diagnose PE. Using convolutional neural networks (CNN) based deep learning, algorithms becoming accessible, detect those lesions [12Arbabshirani M.R. Fornwalt B.K. Mongelluzzo al.Advanced machine learning action: identification scans workflow integration.NPJ Digit Med. 1 (Published 2018 Apr 4): 9https://doi.org/10.1038/s41746-017-0015-zCrossref (183) 13Prevedello L.M. Erdal B.S. Ryu J.L. al.Automated test notification system using imaging.Radiology. 285: 923-931https://doi.org/10.1148/radiol.2017162664Crossref (141) 14Chang P.D. Kuoy Grinband J. al.Hybrid 3D/2D network evaluation CT.AJNR 1609-1616https://doi.org/10.3174/ajnr.A5742Crossref (121) multiple roles assurance productivity enhancement. certain within specific yet fully investigated. Implementing tool during stream, react earlier and/or even notice be easily overlooked [15Chilamkurthy S. Ghosh R. Tanamala al.Deep scans: study.Lancet. 392: 2388-2396https://doi.org/10.1016/S0140-6736(18)31645-3Abstract (360) 16Paiva O.A. Prevedello The impact radiology.Radiol Bras. 50: V-VIhttps://doi.org/10.1590/0100-3984.2017.50.5e1Crossref (15) 17Ojeda P, Zawaideh M, Mossa-Basha al. utility learning: bleeds studies. SPIE Medical Imaging, 2019, Proceedings Volume 10949, 2019: Image Processing; 109493J.Google 18Weikert Winkel D.J. Bremerich angiograms AI-powered algorithm.Eur 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref (27) Scholar].Much research recent years focused solutions, indicated sensitivity 0.95, specificity 0.99, (NPV) 0.98 (PPV) overall accuracy detection. Rao applied solution negative-by-report cases. They found false-negative rate at 1.6%, thus could minimizing negatives [[19]Rao B. Zohrabian V. Cedeno P. Saha Pahade Davis M.A. Utility prospective reviewer - unreported 2020 Feb 24].Acad S1076–6332: 30084-30092https://doi.org/10.1016/j.acra.2020.01.035Abstract (22) Weikert high degree CTPAs, balanced 0.93 0.96 [[18]Weikert Also sensitivity, specificity, values compared senior were reported, 0.85, 0.97, 0.97 0.95 respectively Scholar].The purpose this was assess performance commercially available second reader diverse (e.g., emergency, routine, inpatient, outpatient) assessment, processed calculation respectively.Materials methodsThis conducted approval local institutional medical ethics committee waiver informed consent. Our bipartite pathologies: (PE). Each subdivision consisted 4 stages: (1) Dataset collection; (2) data processing automated tool; (3) revision registered certificate added qualification neuroradiology thorax radiology; (4) analysis.Dataset collectionRandom collection performed from consecutive database patients referred department CTPA, blinded regarding antecedents, diagnose, therapy outcome. Both brain lung study, under age 18 excluded. exams pseudo-anonymized, retaining solely code link each report its respective study. Control CTs excluded, resulting eliminating duplicate final cohort unique scans.A total 500 31 days September 1, until October included. varies considerably terms neurologic pathology signs hemorrhage, mass effect, hydrocephalus, suspected infarct, encephalomalacia no evidence disease. Cases also differ markedly attenuation (respectively hypo-, iso- hyperattenuating), size location (epidural, subdural, subarachnoid intraparenchymal). Scans movement artifacts, sloped postoperative remained included represent practice.Secondly, we considered between July February 1,2020. set consists emboli (central, segmental subsegmental), common diseases interstitial pneumonias, respiratory distress syndrome, sarcoidosis, lymphangitic carcinomatosis, cardiogenic edema normal findings. containing moderate breathing beam hardening well routine practice.CT four different scanners, used (DECT). Table specifies information reference utilized scanner vendor, model, tube voltage, single collimation width, reconstruction slice thickness, kernel doses. Data all axial, coronal sagittal planes, picture archiving (PACS), utilized. DECT scanner, material maps, alternative depicting perfusion defects expert reviewers.Table 1Scanner models, scan- parameters doses indications. Doses median 95% confidence intervals brackets.Scanner modelNumber casesTube voltage (kV)Single width (mm)Reconstructed thickness (mm)Reconstruction methodReconstruction kernelCTDIvol (mGy)DLP (mGy.cm)Intracranial (ICH)500GE Revolution212 (42%)DECT0.6250.625DLIR-MStandard35.8 (35.5–37.2)724 (698–751)GE Discovery 750HD173 (35%)1200.6250.625ASiR 30%Soft38.4 (37.5–39.4)756 (725–787)Philips iCT109 (22%)1000.6250.8iDose3UB (standard)29.9 (29.3–30.6)598 (581–614)Siemens Somatom AS406 (1%)1200.60.6FBPH31s60.81006 (990–1023)Pulmonary (PE)500GE Revolution282 (56%)DECT0.6250.625ASiR-V 70%Standard6.4 (5.9–6.8)229 (211–248)GE 750HD203 (41%)100–1200.6250.625ASiR 30%Detail11.8 (10.8–12.8)441 (405–478)Philips iCT15 (3%)100–1200.6250.9iDose3B (standard)4.1 (3.8–4.4)174 (163–183) Open table tab toolA available, FDA-and CE-cleared (European Devices Directive 93/42/EEC M5) tool, (Aidoc version 1.3, Tel Aviv, Israel) implemented radiological workflow. algorithm trained tested dataset approximately 50,000 [[17]Ojeda Scholar] 28,000 Scholar], collected 9 sites 17 models. According manufacturer’s specifications, acquisition should 64-slice higher reconstructed 0.625 5.1 mm 0.5–3.0 technically inadequate motion severe metal field view sub-optimal bolus (PE).As soon they PACS, automatically pseudo-anonymized subsequently send cloud server. non-enhanced Afterwards, quantitative annotated sent back into PACS additional dicom series. bleeding embolism, these marked series contain arrows pointed situated. seamlessly integrated being typical time (AI PACS) 3 7 min 5–9 studies.Diagnostic reviewFrom presented report. Secondly, evaluated reviewing. original after consensus neuro-radiologists standard. Six board-certified participated 5 up 15 year experience unenhanced reviewers had access prior future studies, able see history reports diagnose. classified true positive, True-positive (TP) contained detected confirmed reviewers. True-negatives (TN) without nor False-positives (FP) defined flagged but negative. False-negatives (FN) decided ICH/PE review. We quantified calculating (NPV), accuracy. concordance pathology, percentage agreement Cohen’s k statistic.In detailed analysis false-positive order identify reason miss classification AI.ResultsTable 2 summarizes From cases, process 77.6% (388 500) No 112 There difference scanners models ranging 84.9% (GE HD) 53.8% (Philips iCT). All 6 Siemens rejected processing. 388 (7.9%) having ICH. Expert 37 (9.5%) hemorrhages. Substantial (kappa-value 0.65) observed. showed 0.84 0.94 specificity. 0.61 respectively. failed label 1.7% (6 337) agreed subspecialists (false tool). Those six summed 3, conclude discrete hemorrhages, subdural hemorrhages parenchymal Twenty labelled review, assigned falcine basal ganglia calcifications (9/20 cases), artifacts (8/20 cases) hyperdense dural sinuses (3/20), shown 3.Table 2Number studied Diagnostic brackets intervals.ICHPENumber presented500500Studies resultAll scanners77.6% (388/500)89.6% (448/500)GE Revolution84.4% (179/212)92.5% (261/282)GE 750HD84.9% (147/173)90.6% (184/203)Philips iCT56.8% (62/109) aSignificantly lower than both GE (p < 0.05, Fischer Exact Probability test).13.3% (2/15) test).Siemens AS400% (0/6) test).N.a.Diagnostic PerformanceSensitivity0.84 (0.68–0.94)0.73 (0.62–0.82)Specificity0.94 (0.91–0.96)0.95 (0.93–0.97)NPV0.98 (0.96–0.99)0.94 (0.91–0.96)PPV0.61 (0.46–0.74)0.73 (0.62–0.82)Accuracy0.93 (0.90–0.96)0.98 (0.96–0.99)N.a. Not available.a Significantly test). 3Detailed AI.Detailed False Negative AISubarachnoid hemorrhages33% (2/6)Subdural (2/6)Parenchymal (2/6)Detailed Positive AIFalcine Basal calcifications45% (9/20)Beam artefacts40% (8/20)Hyperdense sinuses15% (3/20) created 448 (89.6%) CTPA’s. Similar Philips iCT (13.3%), (90.6% 92.5%). 0.73, 0.90 kappa 0.78, indicating substantial concordance. readers 82 did 19 AI). Nine patients, chronic embolisms. masquerading artifacts. underlying concealed present emboli. patient, superimposing vein cause missed embolism. misdiagnosed subsegmental Chronic known central patient. Lobar patient delayed phase.24.4% solution. reviewers, FP due agent-related flow Another masking associated (such infiltrate, metastasis, pleural effusion, atelectasis fibrosis) superposition anatomy (e.g. vein, lymph node, hilar soft tissue, bronchus, azygos artery bifurcation). Withal, caused combination aforementioned factors.Examples Fig. 2, 4.Fig. 2False AI: stroke hemorrhagic transformation identified (white arrow).View Large Figure ViewerDownload Hi-res Download (PPT)Fig. 3Left: (yellow arrow) probably presence surrounded periventricular white matter hypoattenuation. Right: same window/level settings. (For references colour figure legend, web article.)View 4Top (A B): AI, lesion tissue. Bottom AI. (PPT)DiscussionThis two-folded assessed hand, via CTPAs other integration. setting, (388/500) 89.6% (448/500) rates 84.6% (326/385) 91.7% (445/485) if only consider who bulk asses failure follow part protocol. Possible causes hospital-network related attributed to, example, increased noise, few because compliant AI-tool requirements (<64 CT).With value, line work. Previous reported 0.99 NPV PPV remain previous obtained stated above 0.85 accepted findings, amount thereby radiologist’s workload initiation Scholar].False (54% 20/37) negatives, mostly calcifications, streak Roa effortlessly recognized any difficulties An example hypo-attenuation.False occurred limited (6/337) mainly seen small follow-up deteriorating patients.The most commonly sulcal predominant convex regions. less (Fig. 2). Although, finding still explained low HU densities itself, luckily fast diagnosed radiologist. On interpreting scrutinize critic regions carefully. may inconsequential, needs get your attention indicate further examination.With ICH.For achieved rather 0.73 when reached level Firstly, population (9/19 cases). (movement, artifacts). After tool. clarified metastatis, atelectasis, superposing Detailed 4. Examples Although again